System and method for capturing and interpreting images into triple diagrams

ABSTRACT

A machine system includes an image processor to transform a raw digital image to a set of drawing objects comprising individual lines, shapes, and text; a block diagram generator to transform the set of drawing objects into a block diagram; and a flow chart generator to transform the set of drawing objects into a flow chart.

BACKGROUND

Individuals frequently record their thinking in a visual way, in theform of drawings and sketches on a whiteboard or piece of paper,frequently in a group setting. Such recordings are captured in a staticformat, and difficult to memorialize in a way that can be easilymanipulated to add new formatting, record a new thought, correctmisinterpretations, or evolve over time.

Current digital conversion of previously composed, static imagesincluding printed, scanned, hand drawn or non-digitally rendered imagesprovides copies of the images in formats such as bitmap, JPEG, TIFF,PNG, GIFF, RAW, WEBP and other formats. These images are resolutiondependent, preventing the scaling of the images without loss of quality.Vector formats such as CGM, Gerber, SVG, Adobe Illustrator and the likecontain a geometric description which can be rendered smoothly at anydesired scale; however, even these images must be rasterized to bedisplayed on digital monitors. Errors in the digital images cannot beeasily corrected, and when scaled, become more apparent.

Along with scaling issues, the content of the reproduced image cannot beeasily changed or manipulated. Layers may be applied over the digitizedimage, but there are limits as to how much can be accomplished. Forexample, if a page is scanned in at a slight angle, it is difficult ,ifnot impossible, to right the image until it is square on the screen. Ifmaterial is added to the image, or parts of the image are rearranged, itis difficult to have clean breaks, smooth edges, or match font sizes andcolors. There is limited compatability between drawing programs so thatfrequently the only way to correct or alter an image such as a diagramis to rewrite it completely. There is therefore a need for a means toread static image formats and convert them into editable images.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 illustrates an embodiment of a high level system for raw imageacquisition and processing 100.

FIG. 2 illustrates an embodiment of an image processing system 200.

FIG. 3 an embodiment of a raw image processing system 300.

FIG. 4 illustrates an embodiment of an image processor 400.

FIG. 5 illustrates additional details 500 of an image processor 400.

FIG. 6 illustrates an embodiment of an object identification andrefinement sub-system 600 of an image processor 400.

FIG. 7 illustrates an embodiment of a raw image processing system 700.

FIG. 8 illustrates an embodiment of a system for creating a hybrid rawimage 800.

FIG. 9 illustrates an embodiment of a system for the aggregation andprocessing of multiple raw images 900.

FIG. 10 illustrates an embodiment of a system for the processing imagepanoramas 1000.

FIG. 11 illustrates an example of a print or display graph 110.

FIG. 12 illustrates an embodiment of an object oriented drawing triple1200 generated for the print or display graph 110 example illustrated inFIG. 11.

FIG. 13 illustrates an embodiment of a triple diagram generator 1300.

FIG. 14 illustrates a machine system 1400 that may be utilized toimplement components of the described image processing systems.

DETAILED DESCRIPTION Description

A computerized drawing tool includes logic and components capture andconvert a whiteboard/paper/electronic drawing into object definitions.The tool includes logic to recognize annotations on the drawings and tointerpret the annotations into specifically identified signals betweenphysical components, including the timing of those signals, andincluding concurrency if present. The tool includes logic to generate asystem diagram, an action flow diagram, and at least part of a flowchart based on the identified objects, lines, and annotations in thecaptured image. The tool may further comprise logic to identifytransformations as specified by a reference to the component which couldbe an arrow or a line to the component and a label which the labeldescribes the transformation explicitly or by a code or symbol. Based onthe objects (shapes), lines, and annotations on the drawing, the systemcan produce a “triple” set of illustrations: a system diagram showingcomponents and their inter-signaling, an action diagram showing thetiming of signals in the system, and a flow chart illustrating cause andeffect. The system may recognize annotations representing conditions onsignals between components, or conditions on transformations carried outwithin components and caused by signals.

In one embodiment, the system identifies proximity of text to lines thatbegin and end inside or proximate to shapes. The lines are interpretedas signals between components (the shapes). The names of the shapes areidentified from text enclosed by or partially enclosed by the shapes.The signal names are identified from the text proximate to theconnecting lines. The timing of the signals is identified from numbersproximate to the lines, or to the text proximate to the lines.Concurrent signals have the same number. Transformations (actionscarried out in response to received signals by components) areidentified, for example, by text proximate to the end of lines thatdon't end near another component (e.g., “stem” lines) but which do endin or near a component on the other end.

A method of creating an editable image and editable text from ahand-drawn or other static two-dimensional diagram may include receivinga raw image from an image acquisition device; modifying the raw image toa modified image to reduce noise, normalize raw image data, and reducepixels; recognizing horizontal, vertical, and diagonal lines in themodified image using a line recognizer; connecting the lines in themodified image to form connected lines using a connector that detectsand joins proximally positioned terminal ends of the lines in themodified image; recognizing areas bounded by the connected lines asbounded objects using a bounded object recognizer; and/or identifyingand classifying the bounded objects using an object identifier.

Modifying the raw image may include reducing the pixels by reducingpixel information to a single channel to form a reduced raw image,generating a copy of the raw image and applying a smoothing filter tothe copy of the raw image to produce a filtered copy of the raw image,and/or subtracting the filtered copy of the raw image from the reducedraw image to remove a raw image background resulting in pixel setsidentifiable as linear segments.

Modifying the raw image may include determining a starting thresholdbased on an intensity gradient of the raw image, adjusting the startingthreshold to minimize a number of isolated pixels in the raw image,performing a threshold operation to further reduce the pixels, and/orperforming dilation/erosion operations to adjust a thickness ofidentifiable linear segments.

Recognizing the lines in the modified image using the line recognizermay include scanning the modified image signal and identifying astarting pixel based on a particular pixel value; locating an endingpixel connected to the starting pixel by a linear path of pixels equalto a value of the starting pixel; recognizing the linear path of pixelsas a line segment; and/or identifying, locating, and recognizingadditional starting pixels. Recognizing the lines in the modified imageusing the line recognizer may include identifying the ending pixel as anew starting pixel.

Recognizing areas bounded by the connected lines as the bounded objectsthrough the bounded object recognizer may include identifying a firstintersection of two lines within a bounding object as a first vertex,identifying a second intersection of two lines within the boundingobject as a second vertex, calculating an average line length for thelines in the modified image based on a distance between the first vertexand the second vertex, and/or identifying lines that do not form thebounding object as strings.

An editable image and editable text may be generated by transforming ahand-drawn or other static two-dimensional diagram. A hand-drawn orother static raw image is captured using a device for recording visualimages such as a camera, phone, video recorder, scanner, audio recordingdevice, other such device, or a combination thereof. In someembodiments, the images may be recordings with the help of anaccelerometer, gyroscope, or other means for combining motion data withan alignment algorithm. Images may be input from panoramic or standardframe images. In some embodiments, devices may capture multiple framesof the same object, information from which can be aggregated to producethe best manipulable digital raw image. In further embodiments, imagesmay be resident on a computer or have been rendered by a computer orcomputer program.

As individuals frequently doodle or create multiple images on a singlepiece of paper or white board, users can select whether all or part ofthe captured raw image should be converted. In some embodiments, theapplication may zoom in on a particular aspect of a drawing or other rawimage for clarification. In other embodiments, the raw image capturingdevice may show a preview of a raw image to a user so that the user canadjust the input to capture the most accurate raw image for processing.In further embodiments, the raw image may be captured by uploading apreviously created raw image from persistent storage and processed asdescribed below.

The captured raw image is then processed to identify each object in theraw image and determine what each object represents. The objects arethen recreated in digital format and superimposed in a layer over thecaptured raw image so that the user can correct or edit the objects torepresent the desired information or achieve the desired effect. Theopacity of the recreated raw image may be increased or decreased to makethe original raw image more or less visible under the recreated rawimage. This allows a user to easily compare the original static rawimage to the re-created raw image.

Object identification and recreation may be accomplished by processingthe captured raw image to reduce noise, normalize the raw image data,and reduce pixel information using edge detection algorthims. Noisereduction may occur by any means generally used. In some embodiments,noise reduction occurs by the application of one or more filters such asa Gaussian filter, bilateral filter and/or guided filter. Such filtersmay be applied alone, or in combination in any order.

Once the noise in the raw image is reduced as much as possible, theintensity gradient of the raw image is located using one or moreoperators including the Sobel operator, Prewitt operator, and RobertCross Operator. Such operators may be applied alone, or in combinationin any order. The application of these operators allows for theisolation of horizontal, vertical and diagonal lines in the captured rawimage.

Once the raw image gradients are determined, a search is carried out todetermine if the gradient magnitude assumes a local maximum in thegradient direction. Non-Maximum suppression is then applied to removeunwanted pixels and help thin edges. Pixels that belong to an edge arethen identified using thresholding hysteresis. Using thresholdinghystereisis, two thresholds are determined, high and low. A highthreshold is initially applied marking the edges that are more likely tobe certain. The Object recognizer (904) assumes that important edges arealong continuous curves, allowing the capturing of faint sections oflines while discarding background pixels that do not constitute a line.Starting from these and using directional information, edges are tracedthrough the raw image and a lower threshold is then applied, allowingfaint sections of edges to be added to the re-created raw image.

In some embodiments, once edge detection is completed, the raw image maybe further refined using a Hough transform to identify shapes and lines.The Hough transform may be further refined using gradient direction toreduce the number of votes in a parameter space using kernel-based Houghtransform and Hough transform of curves.

Once the captured raw image is processed and the objects in the rawimage are defined, shape recognition is applied. Shape recognition maybe applied by any means generally used. In some embodiments, the numberof continuous lines in a shape may be identified with sharp anglechanges signifying a different line allowing shapes to be identifiedbased on the number of lines in each object. For example, three or morelines is identified as a polygon, 1 line is identified as a circle. Inother embodiments, shape identity can be further refined based on theangles and lengths of the lines to determine more specific shapes suchas squares, triangles, rectangles, rhombuses, or various triangle types.In additional embodiments, shapes may be identified based on patternrecognition. In some embodiments, the user may be presented with optionsfor each object according to degrees of certainty of recognition. Theuser can then select the correct identification to be applied to theobject, may enter the name of a different object, or may edit the shapeof the object to appear as desired. In additional embodiments, the usermay choose to output the diagram in such a hybrid format with part ofthe diagram remaining in the hand drawn or static raw image format.

Any text within the raw image may be processed using optical characterrecognition (OCR). In some embodiments, the processing may include alearning mode to improve its recognition of an individual's handwritingand the shapes they generally draw in a diagram.

Once processed, the digitally generated reproduction of the static rawimage may then be manually edited. Portions of the raw image that couldnot be identified will be correctable by the end user. In someembodiments, the manipulable raw image may be superimposed on thecaptured raw image, allowing the individual manipulating the raw imageto compare the original and the reproduced raw image line by line. Inadditional embodiments, more complex elements may be brought into theraw image in other formats such as as a JPEG, bitmap, TIFF, PNG, orGIFF. The digitally generated reproduction may then be saved inpersistent storage. In some embodiments, the finished reproduced rawimage may be converted to other useful file formats generally used fordisplaying information.

The Overall Processing may Include the Following Relevant Actions:

Prepare the raw image

Find strings and Polygons

Adjust Line Lengths Based on Polygon Sizes

Clean Up strings

Convert String Segments to Lines

Find Arrowheads on the Ends of Lines

Sort Out Chaff from strings

Detect Probable Text Squares

Clean up Polygons into Basic Shapes as Appropriate

Classify Refine Standard Shapes among the Polygons

Find Connected Shapes

Prepare the Raw Image

This is a preliminary step to remove the background (shading,reflections, etc.) from the drawing. The first step is to remove thebackground. This is done by ‘Prepare the raw image’ by the followingsteps:

1. Converting an RGB raw image to greyscale and resizing it to astandard size [previously this was done by the application]

2. Creating a copy (background) of the original raw image and perform awide-window, median-smoothing of the entire copy.

3. Subtracting the background raw image from the original raw image.This removes the background from the raw image leaving the greyscalelines.

4. Calculating a starting threshold based on the rate of change of thenumber of pixels beneath the threshold.

5. Adjusting the threshold value to minimize the number of isolatedpixels (noise). If the threshold is too low, the number of isolatedpixels is high. If the threshold is too high, the number of isolatedpixels is increases due to eroded lines.

6. Performing the threshold operation on the enhanced raw image toproduce a monochrome raw image of lines and background.

7. Performing dilation or erosion operations on the raw image to adjustthe line thickness to an optimized value.

Find Strings and Polygons

This step is performed to identify polygon shapes from lines in a rawimage signal. The raw image is scanned and connected sets of pixels areconverted to connected lines. Connected line sets that form closedshapes are identified as polygons and are removed and added to a polygonset, while connected line sets that do not form closed shapes areunderstood as strings. This is done by ‘Find strings and Polygons’ bythe following steps:

1. Search for white pixels by scanning the raw image from left to rightand bottom to top.

2. When a white pixel is found,

a. Search the surrounding pixels to create the longest straight linepassing through only white pixels. Mark the pixels close to the line asfound.

b. From the start of the line,

i. Search the surrounding pixels to create the longest connectedstraight line passing through only white pixels. Mark the pixels closeto the line as found.

ii. Continue with step (i) until a dead end is reached such as when nomore white or found pixels.

iii. If an intersection is found (two connected lines can be formed)remember the intersection and continue with step (i) in each availabledirection.

c. From the end of the line, perform steps b. (i), b. (ii), and b.(iii).

3. Scan through the strings looking for lines end close to another line.If found, connect them.

4. Find connected lines that form closed shapes (polygons). Extract thepolygons in order of shortest to largest path.

5. Classify the remaining (not closed) connected lines as strings.

Adjust the Line Length

A parameter is used to determine whether a string is a line or not. Thisparameter is adjusted using the average polygon size. The assumption isthat if the average polygon size is large, lines are large and viceversa. This is done by ‘Adjust the Line Length’ by the following steps:

1. Scanning through each poly and calculate the distance between the twofarthest vertexes.

2. Calculating the average of these distances.

3. Adjust the minimum line length parameter based on the average.

Clean Up Strings

Look for strings that are too short or somewhat short and overlay otherstrings and move them to the chaff category. This is done by ‘Clean upstrings’ by the following steps:

1. Scan through the strings.

2. Calculate the length of each string.

3. If the string is shorter than a parameter, move it to chaff.

4. If the string is shorter than another parameter and overlays anotherstring, move it to chaff. This catches strings that are created fromresidue of the original string and polygon search.

Convert Straight Strings to Lines

Looks at each string and see if it qualifies as a line. This is done by‘Convert Straight strings to Lines’ by the following steps:

1. Scan through the strings

2. Find the two vertexes of each string that are farthest apart.

3. Calculate the shortest route between the vertexes.

4. Create a set of points representing the lines along the shortestroute.

5. Fit a line to the set of points.

6. If the line is greater than the minimum line length parameter,

a. Calculate the average error of the points to the line.

b. If the error is less than the minimum line length parameter,

i. Remove the line segments making up the line from the original string.

ii. Move all remaining disconnected segments to new strings (forexample, this might be arrowheads).

iii. Define a line from the best-fit parameters.

iv. Calculate the confidence that this is a line based on the error ofthe vertexes.

v. Add the new line to the lines.

Find Arrowheads

For all lines, look for other lines, strings and chaff to see if theyform an arrowhead. This is done by ‘Find Arrowheads’ by the followingsteps:

1. Scan through the lines.

2. For each end of the line,

a. Search other lines that might form part of the left or right side ofan arrowhead for the line.

b. Search strings that might form part of the left or right side of anarrowhead for the line.

c. Search chaff that might form part of the left or right side of anarrowhead for the line.

3. Validate the arrowheads,

a. Ensure there are two sides to an arrowhead.

b. Make sure the arrowhead lengths are similar.

c. Make sure the angles of the arrow sides are similar relative to theline.

4. If the arrowhead is valid, mark the line as having an arrowhead.

5. Remove the lines, strings and chaff that make up the arrowhead.

Find Strings that Qualify as Chaff

For all strings, look for strings that should be chaff. The originalstrings have been modified by removing polygons, lines. This is done by‘Find strings that Qualify as Chaff’ by the following steps:

1. Scan through the strings.

2. Check each string against other strings to see if it is chaff.

a. Is it isolated from other strings?

b. Is it short?

3. If it is chaff, remove it from the strings and add it to the chaff.

Find Probably Text

Look for strings, lines and chaff that may be text. This is done by‘Find Probably Text’ by the following steps:

1. Assume any strings, lines or chaff that are fully enclosed in apolygon are assumed to be text.

a. Check each line to see if it is fully-enclosed within a polygon. Ifso, add it to the text.

b. Check each chaff to see if it is fully-enclosed within a polygon. Ifso, add it to the text.

c. Check each string to see if it is fully-enclosed within a polygon. Ifso, add it to the text.

2. For each string, examine the bounding rectangle and see if it meetsthe width and height and width-to-height ratios to be text. If so, addit to text.

3. Look for text rectangles that are in close proximity to each otherand combine them as one text object.

4. Scan through the chaff and add any chaff enclosed within a text boxto the text for that text box.

5. Add the text and text boxes to the text collection.

Clean Up Polygons into Basic Shapes as Appropriate

Look at each polygon and see if it is close to a basic shape from atriangle to a pentagon. This is done by ‘Clean Up Polygons into BasicShapes as Appropriate’ by the following steps:

1. Scan through the polygons

2. Calculate the best fit error of 3-sided (triangle) through 5-sided(pentagon) polygons.

3. Select the shape that has the lowest error.

4. If the error is low enough, replace the polygon vertexes with thebest-fit vertexes.

Classify Refine Standard Shapes among the Polygons

Examine each polygon found and see if it is a rectangle, a circle, atriangle, a cloud, or just an n-sided polygon. For each polygon:

1. Check for a 4-sided polygon and classify it,

a. If it does not have 4 line segments, go to step 2.

b. Based on which sides are close to parallel, or close to equal length,classify the shape as a:

i. Rectangle

ii. Parallelogram

iii. Trapezoid

iv. Rhombus

v. Square

vi. 4-sided polygon

c. If any of (i) through (v), adjust the sides of the shape to idealshapes.

d. Calculate the confidence level based on the fit error between theoriginal polygon points and the ideal

e. Save the shape details.

2. Check for a 3-sided polygon and classify it,

a. If it does not have 3 line segments, go to step 3.

b. If the distance of a vertex to the opposite side is too shortrelative to the others, set this to a 3-sided polygon

c. Based on the lengths of the sides adjust the sides and true up theshape to be either a

i. Equilateral Triangle

ii. Isosceles Triangle

iii. General Triangle

d. Calculate the confidence level based on the fit error between theoriginal polygon points and the ideal.

e. Save the shape details.

3. Check if the polygon is a Cloud.

a. If the polygon has less than a parameter number of vertexes, go tostep 4.

b. Calculate the centroid of the vertexes.

c. Calculate the distances from each vertex to the centroid.

d. Start at the closest vertex and find the alternate closest andfarthest vertexes.

e. For each closest-farthest-closest vertexes, calculate how well thevertexes between the two close vertexes fit an ellipse. This fits the“petals” of the cloud.

f. If the error of the fits is greater than a parameter or the number ofpetals is less than a parameter, go to step 4.

g. Calculate the confidence level based on the petal fit error.

h. Save the shape details.

4. Check for an ellipse or circle.

a. Fit the points to an ellipse using the Simplex method fittingalgorithm.

b. Determine the average and maximum error of the fit to the fit points.If the average error is less than a tolerance and the maximum error isless than a given tolerance, mark it as an ellipse.

i. Compare the major and minor axes of the ellipse. If the difference isless than a tolerance,

1. Fit a circle to the fit points.

2. Mark the shape as a circle.

ii. Calculate the confidence based on the fit error.

iii. If the shape is not a circle, check the angle of the ellipse and ifit is close to horizontal/vertical, make it horizontal/vertical.

iv. If the shape is not a circle or ellipse, mark it as an a polygon

c. Save the shape details

Find Connected Shapes

Find lines that appear to connect shapes and mark them with the shapenumbers. This is done by ‘Find Connected Shapes’ by the following steps:

1. For each line, check the end points to see if they are close to theidentified shapes.

2. If the shape is close mark the line with that shape number.

Other Refinements that May be Implemented:

1. Parallax and Tilt—correcting for pictures taken when the cameracharge-coupled device (CCD) is not parallel to the whiteboard.

2. Detection and exclusion of areas that are not part of the whiteboard.

3. Use of color to segment shapes—using colors may help identifyconnected line segments.

DRAWINGS

FIG. 1 illustrates an embodiment of a high level system for raw imageacquisition and processing 100. The system comprises client device 104,machine network 108, and server system 102. The client device 104receives a raw image signal of a print or display graph 110 from thecamera 106 and in response processes the raw image to reduce noise,normalize the raw image data and reduce pixel information. The serversystem 102 receives a processed image from client device 104 and inresponse operates image processor 112 logic (described later) totransform the processed image into a format in which image lines andother geometric objects are manipulable.

FIG. 2 illustrates an embodiment of an image processing system 200. Thesystem comprises camera 106, pixel reducer 204, background remover 206,noise reducer 208, image normalizer 210, image processor 112, and clientdevice 104. The camera 106 captures an RGB (or other color format) imageof the print or display graph 110. The pixel reducer 204 receives thecaptured image from the camera 106 and in response converts the image togreyscale and resizes it to pre-set dimensions. The background remover206 receives the modified image from the pixel reducer 204 and inresponse removes the image background, retaining in the image greyscalelines. The noise reducer 208 receives a modified image from thebackground remover 206 and in response adjusts threshold values tominimize a number of isolated (unconnected) pixel values. The noisereducer image normalizer 210 receives a modified image from the noisereducer 208 and in response performs threshold operations to produce amonochrome image. The image processor 112 receives a modified image fromthe image normalizer 210 and in response performs processing to convertthe image into a format in which image lines and other geometric objectsare manipulable. This final format is communicated to the client device104.

FIG. 3 an embodiment of a raw image processing system 300. The systemcomprises an object recognizer 302, an object identifier 308, andvarious sensors and transducers (e.g., camera 106, sensor/transducer(sound) 304, and sensor/transducer (position) 306).

The object recognizer 302 receives a raw image and readings/data fromsensors and transducers and in response analyzes the raw image, applyingthe readings, generating a list of bounding areas that are determined tocontain atomic objects to be identified and converted in the raw image.The object identifier 308 receives the object bounds formation from theobject recognizer 302 and in response analyzes each object andidentifies the type and/or content of the object.

FIG. 4 illustrates an embodiment of an image processor 400. The imageprocessor 400 (e.g., an embodiment of image processor 112) comprises aconnected line identifier 402, a line connector 404, an objectrecognizer 302, a polygon identifier 406, and a string identifier 408.

A modified image from the image normalizer 210 is received by theconnected line identifier 402 which scans the image for connected setsof pixels that are converted to connected lines. The line connector 404receives lines from the connected line identifier 402 and in responsescans for line ends that are proximate to other line ends (within a setradius, for example) and connects them (adds pixels to the image toconnect the ends, for example).

The object recognizer 302 receives connected lines from the lineconnector 404 and in response generates a list of bounding areas(polygon sets). The polygon identifier 406 receives the polygon set fromthe object recognizer 302 and in response analyzes each polygon objectand identifies the type and/or content of the object. The stringidentifier 408 receives the modified image (including connected lines)from the object recognizer 302 and in response refines the set byidentifying lines and removing strings determined to be chaff.

FIG. 5 illustrates additional details 500 of an image processor 400. Theimage processor 400 comprises object recognizer 302, line lengthidentifier 504, line length adjuster 502, straight string converter 506,separator 508, and shape connector 510.

The object recognizer 302 provides polygon sets to the line lengthadjuster 502 and the line length identifier 504. The line lengthadjuster 502 determines a minimum line length parameter based on anaverage polygon size. The line length identifier 504 receives aprocessed image from the object recognizer 302 and in response scansthrough strings and calculates the length of each string. The linelength identifier 504 receives a line length parameter from the linelength adjuster 502 and in response compares the length of each stringand compares it to the line length parameter, and moves a string to achaff set if string is shorter than the parameter and overlays anotherstring.

The straight string converter 506 receives an image from the line lengthidentifier 504 and in response scans through the strings in the imagedata and calculates the shortest path between furthest apart vertexes ofthe strings. The straight string converter 506 receives a line lengthparameter from the line length adjuster 502 and in response compares theshortest path distance to minimum line length parameter if string is astraight line.

The separator 508 receives a string set from the straight stringconverter 506 and in response ignores modified strings (straight line,polygon, etc) and compares strings against themselves to determineremoval as chaff. The shape connector 510 receives a processed imagefrom the separator 508 and in response combines the processed image datacomprising modified strings with a processed polygon set to generate astructured drawing.

FIG. 6 illustrates an embodiment of an object identification andrefinement sub-system 600 of an image processor 400. The polygon refiner602 receives a polygon set signal from the object recognizer 302 and inresponse analyzes each polygon for similarity to a basic shape from atriangle, quadrilateral, or pentagon set, forming a refined polygon set.The polygon identifier 406 receives a refined polygon set from thepolygon refiner 602 and in response examines each polygon in the set formore specific shape typing to a rectangle, a circle, a triangle, acloud, or just an n-sided polygon. The shape connector 510 receives aprocessed polygon set from the polygon identifier 406 and in responseidentifies lines that connect polygons, and tags them with numbers,generating a manipulable image. The 410 also receives a processed imagefrom the string identifier 408 and in response applies the refinedpolygon set to find lines that connect polygons, and to tag them withnumbers and generating a manipulable image.

FIG. 7 illustrates an embodiment of a raw image processing system 700.The raw image processing system 700 converts a print or display graph110 including geometric object representations into a format in whichthe geometric objects are parameterized (converted to structuredmetadata defining the geometry of the objects) in a structured machinedata graph. The raw image processing system 700 comprises camera 106,image processor 112, persistent machine storage 702, persistent machinestorage 704, location sensor 706, preview application 708,object-oriented drawing editor 710, format converter 712, and digitalfile 716.

The image processor 112 receives an image from the camera 106 andpositional data from the location sensor 706 and in responsecross-references the positional data with the raw image to generatestructured drawing data. The raw image may be provided to the imageprocessor 112 from persistent machine storage 702.

The preview application 708 receives the structured drawing data fromthe image processor 112 and in response presents it to the user 714 forpreview, allowing the user to continue capturing raw image data in anattempt to obtain more accurate structured data, or accepting thepreview and continuing. The object-oriented drawing editor 710 receivesstructured drawing data from preview application 708 and in responsedisplays it to the user along with editing tools to change thestructured drawing data (move objects, connect objects, resize objects,etc.).

The user 714 may operate the object-oriented drawing editor 710 to savethe modified data to digital file 716 on the persistent machine storage704 for later use, potentially converting the structured drawing data toa different file format in the process by engaging the format converter712.

FIG. 8 illustrates an embodiment of a system for creating a hybrid rawimage 800. The system for creating a hybrid raw image 800comprisescamera 106, image processor 112, user input device 802, andobject-oriented drawing editor 710.

The image processor 112 receives a raw image from the camera 106 and inresponse processes the raw image into a manipulable raw image. Theobject-oriented drawing editor 710 receives a processed (manipulable)raw image signal from the image processor 112 and in response acceptsinputs to produce overlay data and manipulations in the raw image. Theobject-oriented drawing editor 710 receives overlay data from the userinput device 802 and in response applies the overlay data (which caninclude movement, re-sizing, deletions, additions, etc.) to the rawimage to produce a compound (hybrid) raw image object in machine memory.

FIG. 9 illustrates an embodiment of a system for the aggregation andprocessing of multiple raw images 900. The system for the aggregationand processing of multiple raw images 900 comprises camera 106,aggregator 902, and image processor 112.

The system utilizes multiple images, from possibly multiple vantagepoints/perspectives, to identify objects and interconnections amongobjects in a print or display graph 110. The aggregator 902 receives andcombines multiple raw images and transforms them into an aggregate rawimage (using known image combination techniques to enhance resolutionand image quality). The image processor 112 receives the aggregated rawimage from the aggregator 902 and in response processes the aggregateraw image in the manners already described for a single raw image.

FIG. 9 illustrates an embodiment of a system for the processing imagepanoramas 1000. The system for the processing image panoramas 1000comprises a camera 106, panorama renderer 1002, image processor 112, anduser input device 802.

The system utilizes a panorama, representing multiple images frommultiple vantage points/perspectives stitched together into a singleimage, to identify objects and interconnections among objects in a printor display graph 110. The panorama renderer 1002 receives an imagepanorama and renders it for a user to select the best portion(s) toconvert into a manipulable set of drawing objects. The image processor112 receives the aggregated raw image from the panorama renderer 1002and in response processes the selected raw image portion(s) in themanners already described for a single raw image.

FIG. 11 illustrates an example of a print or display graph 110. Theprint or display graph 110 may for example be a whiteboard or papersketch which may be processed into a ‘triple’ set of computerizedillustrations (system diagram, action flow diagram, and flow chart). Thesketch includes shapes (e.g., circles) representing components in amachine system. There are lines between the shapes, annotated with textand numbers. There are lead lines with proximate text terminating withinthe shapes, representing processing/transformations by the components.

FIG. 12 illustrates an embodiment of an object oriented drawing triple1200 generated for the print or display graph 110 example illustrated inFIG. 11.

The image processing system 200 is operated to identify shape objects inthe print or display graph 110 and communication lines (e.g., arrows)between the shape objects. Text strings associated in the print ordisplay graph 110 with the shape objects (via position and proximity,for example contained by the shape objects) are identified, and graphicstencils having a configured correspondence with the shape objects areselected to represent the corresponding shape objects. The text stringsin the print or display graph 110 are then made labels to the stencils,and connector objects (e.g., arrow connector objects) are insertedbetween the stencils corresponding to the identified communication linesin the print or display graph 110. The result is a connected system(i.e., block) diagram as illustrated for example by system block diagram1204.

Stem lines (e.g., lines having a floating endpoint proximate with text,and a second endpoint terminating on, in, or near a shape object)areidentified and the associated text is made into actions for blocks in aflow chart (flow chart 1206). The top to bottom order for the actions inthe flow chart is determined by identifying a sequence number proximatewith or otherwise associable with the communication lines in the printor display graph 110.

A data flow diagram (data flow diagram 1202) may also be generated fromthe print or display graph 110. Vertical lines in the data flow diagramare labeled with the stencil (shape object) labels, and horizontal(communication) lines are labeled with labels identified as associatedwith communication lines in the print or display graph 110.

The system may auto-generate text to associate with the object orienteddrawing triple 1200, by traversing the generated data flow diagram 1202,flow chart 1206, and system block diagram 1204. The traverse textcombines the actors (shape labels), actions (stem line labels), andcommunications (communication line labels) forming the system into anatural language set of one or more sentences. The sentences are orderedaccording to the sequence numbers associated with the communicationlines.

An example of auto-generated traverse text that would be generated andassociated with the object oriented drawing triple 1200 is: “The systemcomprises a television, a first mobile device, a server, and a secondmobile device. The first mobile device receives a channel selectionsignal from the television and in response matches the channel selectionto an entry in an EPG. The server receives a content title signal fromthe first mobile device and in response correlates the content title toa stream file(s). The second mobile device receives a stream signal fromthe server and in response renders the stream for viewing.”

FIG. 13 illustrates an embodiment of a triple diagram generator 1300.The triple diagram generator 1300 comprises image processor 112,stencilizer 1302, block diagrammer 1308, line characterizer 1304, flowcharter 1310, label characterizer 1306, and data flow diagrammer 1312.

The image processor 112 outputs a set of drawing objects (lines, textstrings, and shapes for example) generated by transforming a print ordisplay graph 110. All or subsets of the drawing objects are received bythe stencilizer 1302 (which is optional in the triple diagram generator1300), the line characterizer 1304, and the label characterizer 1306.

The stencilizer 1302 accesses a database (e.g., database 1414) to locatestencils corresponding to drawing objects. The correspondence betweenstencils and drawing objects may be based upon text strings that thesystem identifies as being labels for the drawing objects. Thus, thestencilizer 1302 may also act upon an output of the label characterizer1306.

The line characterizer 1304 analyzes lines that are not part of shapesin the drawing for characteristics indicative of a line type(communication lines, stem lines for example). Lines are tagged(associated with) their characteristic type and other attributes, suchas source and destination objects for communication lines, shape objectbeing labeled for object labels, and shape object actor for action stemlines.

The label characterizer 1306 analyzes the text strings in the drawingfor characteristics indicative of label types (sequence numbers, objectlabels, communication line labels, actions for example) and tags thetext strings with their characteristic type and possible otherattributes (e.g., associated communication line, stem line, or shapeobject).

Tagged labels and tagged lines are input to the data flow diagrammer1312, along with drawing objects, and are transformed into the data flowdiagram 1202. Those skilled in the art will appreciate how this may bedone in a number of ways once the drawing labels and lines areidentified and characterized as described herein.

Tagged labels and tagged lines are input to the flow charter 1310, alongwith drawing objects, and are transformed into the flow chart 1206.Those skilled in the art will appreciate how this may be done in anumber of ways once the drawing labels and lines are identified andcharacterized as described herein.

Tagged labels, stencils, and lines are input to the block diagrammer1308, along with drawing objects, and are transformed into the systemblock diagram 1204. Those skilled in the art will appreciate how thismay be done in a number of ways once the drawing labels and lines andstencils are identified and characterized as described herein. Thestencilizer 1302 may not be utilized in all embodiments, in which casethe shapes from the transformed raw image will be used instead in thesystem block diagram 1204.

FIG. 14 illustrates a machine system 1400 that may be utilized toimplement components of the described image processing systems. Forexample, the machine system may implement all or portions of the imageprocessor 112.

In various embodiments, machine system 1400 may include a desktop PC,server, workstation, mobile phone, laptop, tablet, set-top box,appliance, or other computing device that is capable of performingoperations such as those described herein. In some embodiments, machinesystem 1400 may include many more components than those shown in FIG. 2.However, it is not necessary that all of these generally conventionalcomponents be shown in order to disclose an illustrative embodiment.Collectively, the various tangible components or a subset of thetangible components may be referred to herein as “logic” configured oradapted in a particular way, for example as logic configured or adaptedwith particular software or firmware.

In various embodiments, machine system 1400 may comprise one or morephysical and/or logical devices that collectively provide thefunctionalities described herein. In some embodiments, machine system1400 may comprise one or more replicated and/or distributed physical orlogical devices.

In some embodiments, machine system 1400 may comprise one or morecomputing resources provisioned from a “cloud computing” provider, forexample, Amazon Elastic Compute Cloud (“Amazon EC2”), provided byAmazon.com, Inc. of Seattle, Wash.; Sun Cloud Compute Utility, providedby Sun Microsystems, Inc. of Santa Clara, Calif.; Windows Azure,provided by Microsoft Corporation of Redmond, Wash., and the like.

Machine system 1400 includes a bus 1402 interconnecting severalcomponents including a network interface 1408, a display 1406, a centralprocessing unit 1410, and a memory 1404.

Memory 1404 generally comprises a random access memory (“RAM”) andpermanent non-transitory mass storage device, such as a hard disk driveor solid-state drive. Memory 1404 stores an operating system 1412.

These and other software components may be loaded into memory 1404 ofmachine system 1400 using a drive mechanism (not shown) associated witha non-transitory computer-readable medium 1416, such as a floppy disc,tape, DVD/CD-ROM drive, memory card, or the like.

Memory 1404 also includes database 1414. In some embodiments, server 200(deleted) may communicate with database 1414 via network interface 1408,a storage area network (“SAN”), a high-speed serial bus, and/or via theother suitable communication technology.

In some embodiments, database 1414 may comprise one or more storageresources provisioned from a “cloud storage” provider, for example,Amazon Simple Storage Service (“Amazon S3”), provided by Amazon.com,Inc. of Seattle, Wash., Google Cloud Storage, provided by Google, Inc.of Mountain View, Calif., and the like.

References to “one embodiment” or “an embodiment” do not necessarilyrefer to the same embodiment, although they may. Unless the contextclearly requires otherwise, throughout the description and the claims,the words “comprise,” “comprising,” and the like are to be construed inan inclusive sense as opposed to an exclusive or exhaustive sense; thatis to say, in the sense of “including, but not limited to.” Words usingthe singular or plural number also include the plural or singular numberrespectively, unless expressly limited to a single one or multiple ones.Additionally, the words “herein,” “above,” “below” and words of similarimport, when used in this application, refer to this application as awhole and not to any particular portions of this application. When theclaims use the word “or” in reference to a list of two or more items,that word covers all of the following interpretations of the word: anyof the items in the list, all of the items in the list and anycombination of the items in the list, unless expressly limited to one orthe other.

“Logic” refers to machine memory circuits, non transitory machinereadable media, and/or circuitry which by way of its material and/ormaterial-energy configuration comprises control and/or proceduralsignals, and/or settings and values (such as resistance, impedance,capacitance, inductance, current/voltage ratings, etc.), that may beapplied to influence the operation of a device. Magnetic media,electronic circuits, electrical and optical memory (both volatile andnonvolatile), and firmware are examples of logic. Logic specificallyexcludes pure signals or software per se (however does not excludemachine memories comprising software and thereby forming configurationsof matter).

Those skilled in the art will appreciate that logic may be distributedthroughout one or more devices, and/or may be comprised of combinationsmemory, media, processing circuits and controllers, other circuits, andso on. Therefore, in the interest of clarity and correctness logic maynot always be distinctly illustrated in drawings of devices and systems,although it is inherently present therein.

The techniques and procedures described herein may be implemented vialogic distributed in one or more computing devices. The particulardistribution and choice of logic will vary according to implementation.

Those having skill in the art will appreciate that there are variouslogic implementations by which processes and/or systems described hereincan be effected (e.g., hardware, software, and/or firmware), and thatthe preferred vehicle will vary with the context in which the processesare deployed. “Software” refers to logic that may be readily readaptedto different purposes (e.g. read/write volatile or nonvolatile memory ormedia). “Firmware” refers to logic embodied as read-only memories and/ormedia. Hardware refers to logic embodied as analog and/or digitalcircuits. If an implementer determines that speed and accuracy areparamount, the implementer may opt for a hardware and/or firmwarevehicle; alternatively, if flexibility is paramount, the implementer mayopt for a solely software implementation; or, yet again alternatively,the implementer may opt for some combination of hardware, software,and/or firmware. Hence, there are several possible vehicles by which theprocesses described herein may be effected, none of which is inherentlysuperior to the other in that any vehicle to be utilized is a choicedependent upon the context in which the vehicle will be deployed and thespecific concerns (e.g., speed, flexibility, or predictability) of theimplementer, any of which may vary. Those skilled in the art willrecognize that optical aspects of implementations may involveoptically-oriented hardware, software, and or firmware.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood as notorious by those within the art that each functionand/or operation within such block diagrams, flowcharts, or examples canbe implemented, individually and/or collectively, by a wide range ofhardware, software, firmware, or virtually any combination thereof.Several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in standard integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and/or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies equally regardless of the particular type of signal bearingmedia used to actually carry out the distribution. Examples of a signalbearing media include, but are not limited to, the following: recordabletype media such as floppy disks, hard disk drives, CD ROMs, digitaltape, flash drives, SD cards, solid state fixed or removable storage,and computer memory.

In a general sense, those skilled in the art will recognize that thevarious aspects described herein which can be implemented, individuallyand/or collectively, by a wide range of hardware, software, firmware, orany combination thereof can be viewed as being composed of various typesof “ circuitry.” Consequently, as used herein “circuitry” includes, butis not limited to, electrical circuitry having at least one discreteelectrical circuit, electrical circuitry having at least one integratedcircuit, electrical circuitry having at least one application specificintegrated circuit, circuitry forming a general purpose computing deviceconfigured by a computer program (e.g., a general purpose computerconfigured by a computer program which at least partially carries outprocesses and/or devices described herein, or a microprocessorconfigured by a computer program which at least partially carries outprocesses and/or devices described herein), circuitry forming a memorydevice (e.g., forms of random access memory), and/or circuitry forming acommunications device (e.g., a modem, communications switch, oroptical-electrical equipment).

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use standard engineering practices to integrate suchdescribed devices and/or processes into larger systems. That is, atleast a portion of the devices and/or processes described herein can beintegrated into a network processing system via a reasonable amount ofexperimentation.

What is claimed is:
 1. A machine system comprising: an image processorto transform a raw digital image to a set of drawing objects comprisingindividual lines, shapes, and text; a block diagram generator totransform the set of drawing objects into a block diagram; and an flowchart generator to transform the set of drawing objects into a flowchart.
 2. The machine system of claim 1, further comprising: an actionflow diagram generator to transform the set of drawing objects into anaction flow diagram.
 3. The machine system of claim 1, furthercomprising: a traverse module to produce traverse text for the blockdiagram and flow chart, and to associate the traverse text with theblock diagram and flow chart in a digital file.